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ABSTRACT 

Described is the system of evaluation used to monitor 
and assess the effectiveness of community service/continuing 
education programs in Florida funded through Title I of the Higher 
Education Act of 1965. It is necessary to first describe the State 
Agency's understanding of program evaluation and indicate the 
influences that contribute to that understanding. Part I indicates 
those influences and deals with the State Agency's understanding of 
the concept of program evaluation from the : standpoint of definitiong: 
purpose, nature, and process. Part 11 describes project evaluation a^ 
it existed prior to the development of the present system, and then 
describes the present system as it appears in the State Agency 
Operation's Hanual. (Author) 
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Introduction 



The purpose of this paper is tp describe the system of evaluation 
currently being used to monitor, and assess the effectiveness of, community ^ 
service/continuing education programs in Florida funded through Title I of 
the Higher Education Act of 1965. In so doing it is necessary to first de- 
scribe the State Agency's understanding of program evaluation, and indicate 
the influences that contributed to that understanding. ^ 

Part I of this paper indicates those influences, and deals with the 
State Agency's understanding of the concept of program evaluation froea Viie 
standpoint of definition, purpose, nature, %n6 process. 

Part II describes project evaluation as it existed prior to the de- 
velopment of the present system; and then describes the present system as it 
appears In the State Agency Operations Manual. 



Part I 

The Concept of Program Evaluation 



What is Program EvaTuabTicn? 

Following are four quotations that are- cfescn'ptive of evaluation. 
"The Process of evaluation is essentially tiie process of 
determining to what ex'tent the educational objectives are 
actually being realized by the program of urriculum and 
instruction," (Tyler: 1949, p. 105) 

•'Evaluation is the determination of the results attained 
by so3ve activity designed to accomplish some valued goal 
or objective." (Suchman: 1967, p. 32) 

"Evaluation. .answers the question; How effective and 
efficient has the program beer, in achieving the desired 
objectives for the student group?" (Kemp: 1971, p. 96) 

"Evaluation is. ...the determination of the extent to which 

an educational objective has been accomplished." (Houle: 1972, p. 231) 

The conclusion reached after reviewing these brief references is that 
evaluation is a process that describes achievement, or lack of achievement, 
relative to stated objectives. This conclusion, although true, is too limited 
and one-dimensionai to define program evaluation. 

Patrick Boyle and Irwin Jahns (1970, p. 70) describe program evaluation 

as: 

"The determination of the extent to which the desired objectives 
have been attained or the amount of movement that has been 
made in the desired direction." 
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Then they go on to explaiTi: 

"After the educator has detenm'ned the extent to which 
program objectives have been accomplished, it is quite 
reasonable to ask: Why were these results attained?' 
How can we attain better results next time?" 

Suchinan, in reference to program evaluation, maintains that: 

"Particular attention should be paid to the analysis of 
process - how and why various aspects of the program 
failed or succeeded." 

Cyril Houle indicates that program evaluation answers these simple 
questions'. 

"Was. . .progress as good as expected? If not. why not? 
If so, why? (1972, p. 55) 

The conclusion gained here is that program evaluation not only answers 
the question, "How well have we (or you) done? but also answers the question, 
"How did we (or you) do this well? Program evaluation is not only a process of 
measuring results (ends) it is also a process of assessing the procedures (means) 
that led to the results. Furthermore, it is explicit that program evaluation is 
made relative to clear, well-stated, and assessable program objectives. 
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Process: How Is It Accomplished ? 

Houle says that program evaluation "includes the two closely related 
processes of measurement and appraisal (1972, p. 231) 

Measurement is a determination by objective means of the extent to 
which objectives have been achieved. 

Appraisal is a determination by subjective means of the extent to 
which objectives have been achieved. 

Houle states that: 

"Appraisal may incorporate the data, provided by measurement, 
but it goes beyond them to sum them up, to reflect about their 
meaning, and to make a final culminating assessment of the 
value of the activity." (1972, p. 182) 

Process: When Is It Accomplished ? 

Worthen and Sanders (1973) point out that program evaluation should be 
both formative and summative in nature. 

Summative evaluation is the evaluation of the completed program, and can 
Include an assessment of both end results and the means that contributed to the 
resuTts. Summative evaluation questions: "How did we/you do?" and "How did 
we/you do it?" Answers to these questions serve to answer other questions and 
make decisions about future programs. Examples are: "Was the program worthwhile? 
"Should it be done again?" and, if so, what changes might be necessary to improve 
outcomes. 

Formative evaluation is on-going^ beginning with the inception of the 
program and addresses the question: "How are we/you doing?" and "How are we 
doing it?" and leads to the question: "What changes are necessary to improve 
outcomes?" 

3 
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Alkin and Htz-Gibbon (1975) say that program evaluation should also 
be pre-fonnative in nature. Pre-fonnative evaluation deals with needs 
.assessment and program planning, and addresses the questions: "What needs 
to be accomplished?" and "How do we go about accomplishing it?" 
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Purpose 

The purpose of sunwative evaluation is the improvement of future 
progranming. The purpose of formative evaluation is the improvement of the 
present program. Banathy indicates the importance of formative evaluation 
when he says: 

"The purpose of evaluation is to ensure that the objectives 

of the system (program) are being met' or, if not, that adjust- 
ments will be introduced in order to correct the system so that 
objectives can be eventually attained." (1968, p. 79) 

Suchman stresses the importance of a formative system of evaluation thi 
is built-in to the program (1967, p. 134) and maintains that such a system 
may serve the following valuable functions for program operation. (1967, 141) 

1. Determine the extent to which program activities 
are achieving the desired objectives. Measure 
the degree of progress toward goals 

and indicate level of attainment. 

2. Point out specific strong and weak points of 
program operation and suggest changes and modi- 
fications of procedures and objectives. Increase 
effectiveness by majtimizing strengths and 
minimizing weaknesses. 

3. Examine efficiency and adequacy of programs compared 
to other methods and total needs. Improve program 
prQce<lures and increase scope. 

5 
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4. Provide quality-controls. Set standards of performance and 
check on their continuous attainment. 

5. Help to clarify program objectives by requiring operational 
definition In terms of measurable criteria. Challenge the 
"taken-for-granted" assumptions underlying progi'i?ns. Point 
out Inconsistencies in objectives or activities. 

6. Develop new procedures and suggest new approaches and pro- 
grams for future proyrams. , . 

7. Provide checks on possible "boomerang" or negative side 
effects. Alert staff to possible changes of the program. 

8. Establish priorities among programs in terms of best use 
of limited resources-funds, personnel, and time. 

9. Indicate degree of transferability of program to other 
areas and populations. Suggest necess$yy modifications 
to fit changing times and places. 

10. Advance scientific knowledge base of professional practice 
by testing effectiveness of proposed preventive and treat- 
ment programs. Suggest hypotheses for future research. 

n. Advance administrative science by testing effectiveness of 
different organizational structures and modes of operation. 

12. Provide public accountability. Justify program to public. 
Increase public support for successful programs and decrease ■ 
demand for unnecessary or unsuccessful ones. 

13. Build morale of staff by involving them in evaluation of their 
efforts. Provide goals and standards against which to measure 
progress and achievement. 
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14. Develop d critical attitude among staff and field personnel. 
Increase comnunication and information among program staff 
resulting in better coordination of services, 

Alkin and Fitz-Gfbbon say that pre-formative evaluation allows 
the evaluator to provide information on "the perceived importance of 
relevant goal areas, their current status, and the relative priorities of 
each." During the program planning, the evaluator can provide information 
about "competing educational programs that might be utilized for achieving 
the desired goals/* as well as provide evaluative feedback on the planning 
document Itself. (1975, p. 3) 
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Smmiary of Part I , 

A review of the literature has resulted in the following conclusions 
regarding Program Evaluation. 

Definition: 

Program evaluation is an assessment of progress relative to 
stated objectives, as well as an identification of reasons 
why the program has. or has not. achieved those objectives. 

Purpose : 

The purpose of program evaluation is program improvement, 
as well as accountability. 

Nature: 

Program evaluation can be pre-formative. formative, or, 
sunsnative in nature. Summative evaluation is useful in 
planning '^Hture programs. Pre-formative evaluation is useful 
in improvinij the program plan. Formative evaluation is useful 
in improving i;Ve current program. 

Process; 

Program evaluatif; i involves measuring and assessing results (ends) 
and the means rp^iponsible for those results. 

It is the as- ^rtv^l^ of this paper that these conclusions are descriptive 
of effective prcs.dm evaluation. 



Part 11 

Evaluation of HEA Title 1 Projects 
.in Florida 
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Evaluation of Title I HEA Projects In Florida : 
The Present State of Affairs '' 

FederaVregulatlons for Title I HEA stipulate that: 
"The State Plan shall contain a statement of the policies 
and procedures to be followed by the State Agency in making 
periodic, systematic, and objective administrative reviews 
and evaluations in order to evaluate the status and progress 
of particular programs in terms of the annual program proposals 
and overall objectives stated In the plan." (Federal Register; 
Vol. 31, Mo. 68) 

A review of the State Plan and the State Agency Operations Manual 
reveals that the State Agency has complied with the Federal Regulation by 
establishing the following methods of project evaluation. 

1. Proposal Evaluation- performed by the screening, or review 
committee. (The State Advisory Council) 

2. Progress Report- a questionnaire (developed by the State 
Agency) which is completed by each Project Director and 
submitted to the State Agency at a mid-point in the 
life of the project. 

3. On-site visitation- performed by the State Agency at any 
point during the life of the project. 

4. Final Report- a questionnaire (developed by the Federal 
Agency) which is completed by each Project Director and 
submitted to the State Agency upon the termination of the 
project* 



It is important to note that these are only methods of evaluation, 
such as those described by Ray Bard (1971). fhe State Plan or the Operatic 
Manual describes no system of project evaluation that explicitly: 

Defines program evaluation. 

States the purpose of program evaluation. 

Describes the nature of program evaluation, and 

Describes a process foi' program evaluation. 

Such a systeni (The Desired State of Affairs) should be developed by 
the State Agency, written into the Operations Manual, and disseniinated to 
all project directors -nd institutions involved in the Title I HEA Program 

in Florida. 

A model for such a system follows: 
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CHAPTER 8. EVALUATION 
8.1 Project Evaluation 

8.1.1. Definition 

Project evaluation is an assessment of progress 
relative to the project's stated objectives, 
and bn identification of reasons why the project 
has, or has not, achieved those objectives. 

8.1.2 Purpose 
The primary purpose of project evaluation is 
project improvement. A secondary purpose is 
to insure the accumulation of data necessary 
to complete the annual report of the State 
Agency. 

8.2.3 Nature 
The nature of project evaluation is pre-formatlve, 
formative, and summatlve. 

a. Pre-fonnative evaluation will allow the program 
planner to make whatever adjustments are necessary 
to improve the program plan. 

b. Formative evaluations will allow the project staff 
to make whatever adjustments may be necessary for 
project improvement^ 

c. Summatlve evaluation xrlll allow the State Agency 
to measure program effectiveness, and collect the 
data necessary for completion pf the annual report 
pf the State Agency. 

.8.1.4 Process 

The process of project «Vjal.u«$jpn will Involve ineasuring and, 
11 



assessing project results relative to the 
stated objectives, and the procedures and 
methodologies responsible for those results. 
The process of project evaluation will include 
The following methods: 

a. A proposal review will be accomplished by the 
State Agency, in conjunction with the State 
Advisory Council. This review will insure that 
(T) the objectives of the intended project are 
clearly stated, assessable, and acceptable 

to the State Agency, (2) that the procedures 
and methodology described are clearly stated, 
relative to project objectives, and acceptable 
to the State Agency, and (3) a method of evaluation 
is described that will allow the project staff to 
assess the effectiveness of the learning experiences 
delivered by the project. 

b. An interim report ^i^^^l^ t»3 completecl by the project 
director and subfnitted to the State Agency at a 
date specified by the State Agency, (r'^proximately 
mid-point in the life of the project). The 'ormat 
of the interim report will be developed by the 
State Agency, and will be constructed so as to 
allow the project director to describe progress 
toward stated objectives, and identify reasons why 
progress has, or has not, been achieved. 

c. A site visit will be made i;>y the State Agency no 
later than six weeks after receipt of the interim 
report. The site visit will allow the project 

12 
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director and state agency to thoroughly review 
project achievement or non-achievement, discuss 
the reasons for achievement, or non-achievement, 
and develop adju3t5r,ents that may be necessary 
for project improvement. 

As a result of the site visit the State Agency 
win prepare a memorandum for the project director 
which reviews their conference a'jd clearly states 
the adjustments that they both feel are necessary 
for project improvement. This memorandum will be 
sent to the project director no later than one week 
after the site visit. A copy of the memorandum 
will be sent to the president of the institution 
sponsoring the project. 

A final report will be completed by the project 
director and submitted to the State Agency no 
later than two weeks after termination of the 
grant. The Vormat for the final report is 
developed by the Federal Agency and allows the 
project director to make a final description 
of achievement relative to project objectives 
and identify reasons for achievement, or non- 
achievement. 

The final report also provides the State Agency 
with the data and infonmition necessary to 
complete the annual report of the State Agency. 
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System For Project Evaluation 
Title I HEA 



Proposal Review and Evaluation: 
Pre-formative evaluation of: 

bi Procedures for^attaining objectives (means) 
c> Method of evaluating a and b 



• Interim Report: Formative evaluation / 
interim «epo^^^^^^ ^^^^^^^ toward objectives ends) 

b) An assessment of program procedures (means) 

c) A description of method of evaluating a and b 



Site Visitation: Formative evaluation, to be: 

a) Made within six weeks of interim report 

b) An in-depth discussion of a, b, & c, of interim report 
c The basis of program adjustment, if necessary 



'''^'\fZl\ sSro^ro^^ toward objectives (ends) 
-^a] ^^^^^pJ^ofo^M^ 
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Review of the Literature 



The following references and authors influenced the development- 
of this paper: 

Two ^oks, Evaluative Research: Principles and Practice 
in Public Service and Social Action Programs by Edward Suchman, 
and Educational Evaluation: Theory and Practice by Blaine Worthen 
and Jame*- Sanders, contributed to an improved understanding of 
the concept of evaluation. 

Basic Principles of Curriculum and Instruction by Ralph Tyler, 
Instructional Design by Jerrold Kemp, and Instructional Systems 
by Bel a Banathy were all useful in examining evaluation as a 
component of instructional design. 

The Design of Education by Cyril Houle, and "Program Development 
and Evaluation" by Patrick Boyle and Irwin Jahns ( Handbook of 
Adult Education , 1970) dealt with evaluation as a component of 
adult education programming. 

Jack Rotlimari, ir> Planning and Organizing for Social Change , and 
Ronald Havelock, in The Change Agents Guide to Innovation in 
education address the purpose of evaluation in conmunity de- 
velopment progranining. 

Burton Friedman and Laird Dunbar, in Grants Management in Education : 
Federal Impact on State Agencies discuss evaluation of federal 
grant programs. 
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Program and Staff Evaluation by Ray Bard. The Modern Practice 
of Adult Education and Higher Adult Education in the United 
States by Malcolm Knowles. and Management By Ob.iectives In 
Higher Education , edited by C. P. Heaton. were all useful in 
reviewing methods and procedures of program evaluation. 

"Methods and Theories of Evaluating Programs." an article by 
Marvin Alkin and Carol Fitz-Gibbons. defines pre-formatlve 
evaluation, and describes Its purpose In program planning. 
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Afterword 



When this paper was written, the system of evaluation described was 
a proposed system, a deiired state of affairs. This system is now a present 
state of affairs, and therefore the subject of state agency scrutiny and, yes, 
evaluation. Hopefully this scrutiny and evaluation will indicate yet another 
desired state of affairs in program evaluation toward which we may strive. 

So far, (we are presently in the first year of implementation), the 
response from project directors, as well as deans and directors of community 
service and continuing education has been positive and enthusiastic. However, 
we can safely say that the state agency has benefited as much, or more, than 
anyone In the field. A systematic and ordered approach to program evaluation 
has given us new Insights Into the projects we administer. This has resulted 
In an improved understanding of their problems and a greater awareness of the 
many outstanding programs that are addressing community problems of Florida 
fn a significant way. 
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